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Abstract 


This  research  attempts  to  use  the  Navy  Enterprise  Open  Architecture 
Assessment  Tool  (OAAT)  findings  as  a  method  for  analyzing  the  US  Navy’s 
implementation  of  a  Modular  Systems  Approach  (MOSA)  in  its  weapon  systems 
acquisition  programs. 

The  purpose  of  this  research  paper  to  provide  a  preliminary  analysis  of 
assessments  conducted  on  Navy  weapon  system  acquisition  programs  using  the 
Navy  Enterprise  Open  Architecture  Assessment  Tool  (OAAT).  The  OAAT  is  used  to 
assess  a  weapon  system’s  “degree  of  openness”  in  terms  of  the  open  architecture 
maturity  of  that  specific  weapon  system  program  and  its  systems.  Openness  refers 
to  both  business  and  technical  characteristics  of  weapon  systems  that  support 
modular  design,  interoperability,  and  commercial  standards.  A  higher  degree  of 
openness  both  supports  weapon  system  programs  in  terms  of  competition  for 
development  and  support,  as  well  as  facilitates  rapid  technology  insertion.  Although 
the  number  of  weapon  system  programs  and  system  assessments  analyzed  in  this 
research  are  minimal,  this  paper  does  provide  some  preliminary  conclusions  on  the 
Navy’s  implementation  of  a  Modular  Systems  Approach  (MOSA)  in  its  weapon 
systems  acquisition  programs. 

Keywords:  assessment,  modular  open  systems  approach,  OAAT,  open 
systems 
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I 


Introduction 


This  research  continues  the  exploration  of  the  use  of  the  modular  open 
systems  approach  (MOSA)  as  a  method  for  implementing  an  evolutionary 
acquisition  strategy  in  Department  of  Defense  (DoD)  programs.  The  background  on 
the  initial  DoD  and  Navy  policy  on  using  a  MOSA  approach  in  defense  acquisition  is 
presented,  followed  by  a  review  of  the  initial  research  findings.  A  discussion  is  then 
provided  on  the  Navy’s  method  for  assessing  its  implementation  of  a  MOSA 
approach  in  its  acquisition  programs.  This  discussion  will  focus  on  the  use  of  the 
Naval  Enterprise  Open  Architecture  Assessment  Tool  (OAAT).  The  primary  purpose 
of  this  continuing  research  is  to  provide  a  preliminary  analysis  of  the  results  of  the 
OAAT  assessment  of  selected  Navy  acquisition  programs. 
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II 


Background  on  MOSA  Policy 


DoD  5000.1  states  that,  “a  modular  open  systems  approach  shall  be 
employed  where  feasible”  (Under  Secretary  of  Defense  (AT&L),  2003,  May  12a; 
2003,  May  12b).  Furthermore,  in  April  2004,  the  USD  (AT&L)  issued  a 
memorandum  stating,  “all  programs  subject  to  milestone  review  shall  brief  their 
program’s  MOSA  implementation  status  to  the  Milestone  Decision  Authority  (MDA) 
to  determine  compliance”  (Under  Secretary  of  Defense  (AT&L),  2004,  April  5).  This 
USD  (AT&L)  memo  also  directs  program  managers  to  brief  the  result  of  their 
Program  Assessment  and  Rating  Tool  (PART)  assessments  at  all  major  milestone 
and  program  reviews  (Under  Secretary  of  Defense  (AT&L),  2004,  April  5). 

Later  that  year,  the  Office  of  the  USD(AT&L),  Director  of  Defense  Systems, 
issued  instructions  for  MOSA  implementation  and  identified  the  Open  System  Joint 
Task  Force  (OSJTF)  as  the  DoD  lead  for  MOSA.  This  memo  also  identified  MOSA 
as,  “an  integral  part  of  the  toolset  that  will  help  DoD  achieve  its  goal  of  providing  the 
joint  combat  capabilities  required  in  the  21st  century,  including  supporting  and 
evolving  these  capabilities  over  their  total  life-cycle”  (Under  Secretary  of  Defense 
(AT&L),  2004,  July  7). 

In  addition,  in  August  2004,  Assistant  Secretary  of  the  Navy  (Research, 
Development  &  Acquisition)  (ASN  (RDA))  issued  a  policy  statement  that  developed 
a  single  Navy-wide  Open  Architecture  to  account  for  Surface,  Air,  Submarine,  C4I, 
and  Space  domain  unique  requirements.  That  memo  also  assigned  PEO  IWS 
overall  responsibility  and  authority  for  directing  the  Navy's  OA  Enterprise  effort.  An 
OA  Enterprise  Team  comprised  of  OA  domain  leads,  ASN,  OPNAV,  and  SYSCOM 
representatives  was  chartered  and  led  by  PEO  IWS.  The  Team  collectively  oversees 
the  development  and  implementation  of  the  processes,  business  strategies,  and 
technical  solutions  which  support  cross-Enterprise  requirements  in  addition  to 
domain-specific  needs.  The  Enterprise  Team  will  also  define  an  overarching  OA 
acquisition  strategy  and  develop  guidance  that  addresses  incentives,  intellectual 
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property  issues,  contracting  strategies  (i.e.,  integrator's  vs.  prime's),  and  funding 
alternatives  (ASN  (RD&A),  2004). 


Finally,  in  a  23  December  2005  letter,  Deputy  Chief  of  Naval  Operations 
(Warfare  Requirements  and  Program)  established  the  Navy-wide  requirement  for 
OA  and  laid  out  the  priorities  on  which  it  wants  Naval  OA  to  focus.  The  letter, 
“establishes  the  requirement  to  implement  Open  Architecture  (OA)  principles  across 
the  Navy  Enterprise.”  It  establishes  the  OA  Council  (OAC)  of  representatives  of 
N6/N7  Division  Directors  to  work  with  the  OAET  on  the  requirements.  The  letter 
directs  the  OAC,  PEO  IWS  7.0,  and  the  OAET  to  focus  assessment  priorities  in 
support  of  the  following  capabilities:  Track  management,  Combat  ID  (CID),  Data 
fusion,  Time-critical  Targeting  &  Strike,  and  Integrated  Fire  Control  (IFC). 
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Ill 


Initial  Research  Findings 


The  purpose  of  the  initial  MOSA  research  was  to  explore  both  the  use  of  the 
modular  open  systems  approach  (MOSA)  as  a  method  for  implementing  an 
evolutionary  acquisition  strategy,  as  well  as  the  implications  of  using  such  an 
approach  on  the  contracting  process. 

Although  the  phases  of  the  contracting  process  are  the  same  for  MOSA- 
based  program  as  they  are  for  non-MOSA-based  programs,  this  research  found  that 
the  specific  activities  conducted  and  documents  developed  during  the  execution  of 
these  contracting  phases  have  a  direct  influence  on  the  success  of  a  MOSA-based 
program.  For  example,  the  various  options  for  allocating  roles  and  responsibilities 
between  the  government  and  the  contractor  for  the  various  steps  in  the  acquisition 
process  will  influence  the  amount  of  “openness”  in  the  program  and  the  contractor’s 
motivation  for  meeting  the  desired  level  of  openness. 

This  research  indicated  that  the  greater  degree  of  jointness  in  acquisition 
roles  and  responsibilities,  as  well  as  the  greater  degree  of  contractor-developed 
acquisition  documents,  led  to  a  higher  level  of  openness. 

This  initial  research  also  identified  early  involvement  and  participation  by 
industry  in  developing  requirements  and  acquisition  strategy  as  key  factors  in 
successful  MOSA-based  programs.  Program  offices  managing  a  MOSA-based 
programs  should  conduct  extensive  market  research  and  industry  conferences  to 
achieve  this  contractor  involvement.  A  best-value  contract  strategy  that  is  tailored  to 
emphasize  technical  performance  in  open-based  systems  and  COTS  systems  is 
also  a  critical  factor  in  meeting  higher  levels  of  openness  in  MOSA-based  programs. 
A  contract  strategy  which  involves  developing  source-selection  evaluation  factors 
specifically  weighted  to  emphasize  an  open  systems  approach  will  be  critical  for 
MOSA-based  programs. 
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The  structure  of  the  contract  of  a  MOSA-based  program  is  as  important  as  its 
acquisition  strategy. 

This  research  identified  the  use  of  incentive-fee,  award-fee,  and  award-term 
contract  incentives  as  integral  to  the  success  of  MOSA-based  programs.  These 
incentives,  if  structured  appropriately,  are  effective  tools  for  motivating  and 
incentivizing  contractors  to  achieve  higher  levels  of  openness  in  the  design  and 
development  of  systems. 

Finally,  the  consistent  and  aggressive  use  of  the  contractor  past-performance 
information  system,  as  well  as  the  development  and  establishment  of  lessons- 
learned  programs  and  best  practices  will  be  essential  as  more  and  more  MOSA- 
based  programs  are  initiated.  As  contractors  performing  work  on  MOSA-based 
programs  begin  to  realize  that  the  DoD  is  insistent  on  using  open  systems  in 
developing  its  major  weapon  systems,  they  should  begin  to  dedicate  the  required 
resources  to  this  method  of  developing  weapon  systems. 
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IV.  Internal  Assessment  of  MOSA  Implementation 


The  focus  of  this  follow-on  research  is  to  explore  the  effectiveness  of  the 
implementation  of  MOSA  in  Navy  acquisition  programs.  This  research  will  include 
investigating  the  results  of  MOSA-internal  assessments,  specifically  the  results  of 
the  Navy’s  Open  Architecture  Assessment  Tool  (OAAT).  The  results  of  this 
research  will  prove  beneficial  to  senior  Navy  officials  by  providing  data  points  on 
MOSA  implementation  by  analyzing  the  consistency  of  MOSA  compliance  status 
and  internal  assessments  for  specific  Navy  acquisition  programs. 

A.  Open  Architecture  Assessment  Tool 

The  Open  Architecture  Assessment  Tool  (OAAT)  is  a  tool  designed  to  assist 
Navy  program  managers  in  assessing  the  degree  of  "openness"  of  their  programs.  It 
aligns  to  the  Open  Architecture  Assessment  Model  (OAAM)  as  approved  by 
ASN(RDA)  and  provides  a  reproducible  and  objective  method  of  conducting  program 
assessments.  Specifically,  the  OAAT  is  an  analytical  tool  that  evaluates  responses 
to  a  set  of  interrelated  questions  to  provide  program  officers  with  an  objective  and 
evidence-based  assessment  of  the  degree  that  a  program  exhibits  openness  along 
two  axes:  Business/Programmatic  and  Technical.  The  degree  to  which  openness  is 
implemented  is  presented  in  terms  of  business/programmatic  and  technical  criteria. 
The  OAAT  assessment  score  summary  provides  a  summary  of  the  ratings  for  each 
of  the  evaluated  areas  (See  Figure  1). 
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Business  Areas 


Technical  Areas 


Open  Systems  Approach 

Open  Architecture 

Open  Modular  Design 

Interface  Design  and  Management 

Treatment  of  Proprietary  Elements 

Open  Business  Practices 

Peer  Review  Rights 

Technical  Insertion 

Commercial  Standards 

Compliance 


Design  Tenet:  Interoperability 
Design  Tenet:  Maintainability 
Design  Tenet:  Extensibility 
Design  Tenet:  Composability 
Design  Tenet:  Reusability 
General  Design  Tenet 


Figure  1.  Ratings  of  Evaluated  Areas 

B.  Business/Programmatic 

The  business/programmatic  dimension  criteria  include  questions  that 
address:  Open  Architecture,  Modular  Open  Design,  Interface  Design  and 
Management,  Treatment  of  Proprietary  Elements,  Open  Business  Practices,  Peer 
Review  Rights,  and  Technology  Insertion.  The  programmatic  questions  refer  to  the 
processes  and  documentation  employed  to  acquire  and  manage  systems.  The 
programmatic  areas  measured  include  the  following  (Naval  Open  Architecture 
Enterprise  Team,  2006): 

■  Minimization  of  modifications  to  open  standards  that  limit  flexibility 

■  Scope  of  unique  development 

■  Limitation  of  impact  of  proprietary  solutions  on  openness 

■  Requirements  compliance  with  JCIDS 

■  Requirements  compliance  with  Interoperability  and  Supportability 

references 

■  Spiral  development 

■  Exportation  of  reusables,  flexibility/openness 

■  Prime  System  Integrator  competitive  assignment 
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C.  Technical 


The  technical  dimension  criteria  cover  the  essential  OA  design  tenets  of 
Interoperability,  Composability,  Reusability,  Maintainability  and  Extensibility.  The 
technical  questions  refer  to  the  technical  features  of  computing  environments  and 
application  software.  The  technical  areas  measured  are  described  below  (Naval 
Open  Architecture  Enterprise  Team,  2006): 

Interoperability:  How  readily  can  the  program’s  separate  systems  exchange 
information  and  appropriately  utilize  each  other’s  functional  capabilities? 

Maintainability:  What  architectural  characteristics  address  obsolescence  and 
provide  for  timely  technology  refresh,  fixes,  and  upgrades? 

Extensibility:  Does  the  program  follow  a  well-defined  System  Engineering 
process  for  implementing  capability  extension? 

Composability:  Are  the  program’s  systems  capable  of  being  highly  modular 
and  having  minimal  dependencies  (loosely  coupled)  so  they  can  be  readily 
combined  with  other  modules  to  provide  new  types  of  functionality? 

Reusability:  Are  the  assemblies  that  are  candidates  for  reuse  readily 
available,  certified  for  reliability  and  performance,  and  easily  obtained  for 
reuse? 

MOSA:  What  is  the  program’s  level  of  MOSA  Compliance? 

The  answers  to  the  Business/Programmatic  and  Technical  questions  are 
summarized  to  provide  an  overall  score  of  the  percent  total  of  the  answered 
questions  against  the  total  possible  score.  This  is  the  number  that  is  used  to  plot  on 
the  appropriate  dimension  or  axis  of  the  OAAT  matrix,  which  will  be  discussed  next. 


D.  Open  Architecture  Maturity  Matrix 

The  OAAT  provides  an  OA  assessment  matrix  that  displays  the  program’s 
current  state  with  respect  to  business  and  technical  degrees  of  openness.  Each  of 
these  areas  (business  and  technical)  is  rated  on  a  scale  of  0  to  4.  (See  Figure  2.) 
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OAAT  Assessment  Matrix 


Figure  2.  Open  Architecture  Maturity  Matrix 

The  scores  for  these  two  dimensions  are  plotted  on  the  OA  Assessment 
Model,  which  provides  a  graphical  depiction  of  the  current  state  of  OA  maturity  and 
also  identifies  the  progression  towards  higher  levels  of  openness.  The  results  of  the 
OAAT  assessment  are  then  be  used  by  the  program  manager  to  help  improve  the 
program  with  respect  to  Naval  Open  Architecture. 

Using  the  Open  Architecture  Maturity  Matrix,  a  program’s  degree  of  openness 
can  be  rated  using  the  programmatic  and  technical  levels  as  shown  in  Figure  3 
below. 
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Proarammatic  Levels 

Technical  Levels 

4:  Open  and  Net-Centric 

4:  Enterprise 

3:  Common 

3:  Common 

2:  Migrating  to  Openness 

2:  Layered  &  Open 

1:  Connected 

1:  Layered 

0:  Isolated 

0:  Closed 

Figure  3.  Rating  Levels 
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V. 


Assessment  Results 


The  primary  purpose  of  this  continuing  research  is  to  provide  a  preliminary 
analysis  of  the  results  of  the  OAAT  assessment  of  selected  Navy  acquisition 
programs.  This  research  reviewed  the  OAAT  assessment  results  from  two  different 
Navy  domains — Subsurface  and  Air.  It  should  be  noted  that  some  of  these 
assessments  were  conducted  using  the  initial  OAAT  before  the  addition  of  the 
MOSA  PART  questions,  so  some  of  these  scores  may  not  include  the  MOSA 
assessment.  Additionally,  the  names  of  the  specific  programs  and  units  of 
assessment  have  been  deleted  since  they  were  not  cleared  for  public  release. 

A.  Subsurface  Domain 

Figure  4  reflects  the  assessment  results  of  eight  programs  within  the 
Subsurface  domain.  As  illustrated  in  Figure  4,  four  of  the  eight  programs  are  rated 
at  Level  3  or  higher  in  both  the  Technical  and  Programmatic  categories,  reflecting 
High  levels  of  openness.  Additionally,  two  programs  are  rated  at  the  Medium  level 
of  openness,  and  the  remaining  two  programs  are  rated  at  the  Low  level  of 
openness. 
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Sub  Domain  OAAT  Assessments 


20% - 

0 

o%  - - - - - 

0%  20%  40%  60%  30%  100% 

Technical  Levels 

0  1  2  3  4  2 

Figure  4.  Subsurface  Domain  OAAT  Results 

It  should  be  noted  that  some  of  these  assessment  were  conducted  using  the 
initial  OAAT  before  the  addition  of  the  MOSA  PART  questions,  so  some  of  these 
scores  may  not  include  the  MOSA  assessment. 

B.  Air  Domain 

Figure  5  reflects  the  results  of  24  assessments  in  the  Air  Domain.  These 
assessments  were  conducted  on  either  total  aircraft  or  specific  systems. 

In  the  case  of  an  aircraft  assessment,  aircraft  scores  are  either  the  weighted 
average  of  individual  system  scores,  or  they  are  reflective  of  the  overall  avionics 
architecture.  Some  systems  in  aircraft  were  weighted  more  important  for  example, 
the  computing  environment  would  be  weighted  more  important  for  an  aircraft  such 
as  the  E-2  Hawkeye  compared  to  the  H-60  Black  Hawk  helicopter.  In  addition,  not 
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every  system  in  an  aircraft  was  assessed.  The  assessment  decision  was 
determined  by  the  appropriate  program  manager.  Aircraft  with  complex 
architectures  (i.e.  those  with  multiple  missions  or  those  with  multiple  sensors)  were 
generally  broken  out  by  systems,  while  simple  aircraft  architectures  (limited  to  flight 
systems,  navigation,  etc.)  were  assessed  in  their  entirety. 

In  the  case  of  a  system  assessment,  the  systems  selected  for  separate 
assessment  were  determined  by  the  programs  to  be  key  components  of  the  avionics 
architecture.  The  system  scores  were  weighted  and  calculated  to  an  aircraft  score. 
For  instance,  almost  all  aircraft  have  the  same  radio.  It  was  assessed  once,  and  its 
score  was  weighted  based  on  each  aircraft's  architecture  (the  importance  of  the 
radio  to  mission  execution).  The  weighted  value  was  then  averaged  with  other 
components  to  create  the  reported  aircraft  score. 
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Technical  Levels 
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Figure  5.  Aircraft  Domain  OAAT  Results 
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As  reflected  in  Figure  5,  the  majority  of  both  aircraft  and  systems  are  rated  at 
least  at  Level  2  Technical  and  Level  2  Programmatic,  with  almost  half  of  the 
assessments  rated  at  least  Level  3  in  both  Technical  and  Programmatic. 

Of  the  13  aircraft  programs  assessed,  only  4  were  rated  at  Level  3  Technical 
and  Level  3  Programmatic.  The  remaining  9  aircraft  programs  were  rated  at  Level  2 
Technical  and  either  Level  2  or  Level  3  for  Programmatic. 

Of  the  1 1  systems  that  were  assessed  in  the  Air  Domain,  7  were  rated 
between  Level  3  and  Level  4  Technical  and  Level  3  and  Level  4  Programmatic.  The 
remaining  4  systems  were  rated  between  Level  1  and  Level  2  Technical  and 
between  Level  2  and  Level  3  Programmatic. 
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VI 


Analysis  of  Assessment  Results 


Although  data  from  only  two  domains  were  received  in  this  research  (out  of  a 
total  of  six  domains  solicited  for  data — Surface,  C4I,  Space  and  Marine  Corps  did 
not  provide  any  data),  the  assessment  results  can  be  analyzed  to  reach  some 
preliminary  qualitative  conclusions. 

Approximately  half  (16  of  the  32)  of  the  total  assessments  conducted 
(Subsurface  and  Air  Domains)  were  rated  at  the  High  level  of  openness.  Fourteen 
of  the  total  assessments  were  rated  at  the  Medium  level  of  openness,  and  two 
assessments  were  rated  at  the  Low  level  of  openness.  Based  on  these  assessment 
results,  it  can  be  seen  how  the  Open  Architecture  Assessment  Tool  (OAAT)  can  be 
effectively  used  to  assess  a  program’s  (aircraft  or  system)  level  of  openness. 

With  the  assessment  results,  program  managers  can  then  use  the  OAAT  to 
determine  which  categories,  Business/Programmatic  or  Technical,  need  additional 
emphasis  in  order  to  increase  the  program’s  level  of  openness.  For  example, 
programs  that  are  rated  Low  in  business/programmatic  may  need  additional 
emphases  placed  on  using  an  open  systems  approach,  modular  design,  managing 
interface  design,  proper  treatment  of  proprietary  elements,  using  open  business 
practices,  and  so  forth.  On  the  other  hand,  programs  that  are  rated  Low  in  the 
technical  category  may  need  additional  emphasis  placed  on  the  design  tenets  of 
interoperability,  maintainability,  extensibility,  composability,  reusability  and  so  forth. 

Additionally,  the  OAAT  can  be  used,  not  only  for  assessing  levels  of 
openness  on  an  “as  is  “  basis  ,  but  also  for  assessing  a  level  of  openness  on  a  “to 
be“  basis.  That  is,  the  OAAT  can  be  effective  in  developing  a  road-map  for  a 
program’s  strategy  for  achieving  a  specific  level  of  openness  for  the  weapon  system 
or  sub-system.  Through  periodical  assessments,  the  program  manager  can  gage  the 
progress  in  achieving  the  desired  level  of  openness  for  the  program.  Of  course  in 
this  application,  a  program  decision  must  be  made  concerning  the  desired  level  of 
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openness,  given  the  typical  project  constraints  of  cost,  schedule  and  performance. 
Once  that  desired  level  of  openness  decision  is  made,  the  Program  manager  can 
then  use  the  OAAT  and  periodic  assessment  results  as  a  road-map  for  achieving 
that  level  of  openness. 

In  addition  to  supporting  program  managers,  the  OAAT  can  also  support  the 
Milestone  Decision  Authority  (MDAs)  in  assessing  the  program’s  compliance  with 
current  DoD  Open  Systems  policy.  As  directed  by  the  April  2004,  USD  (AT&L) 
memorandum,  “all  programs  subject  to  milestone  review  shall  brief  their  program’s 
MOSA  implementation  status  to  the  Milestone  Decision  Authority  (MDA)  to 
determine  compliance”  (Under  Secretary  of  Defense  (AT&L),  2004,  April  5).  It  would 
seem  that  MDAs  would  want  to  take  advantage  of  the  OAAT  assessment  results  as 
a  gauge  for  determining  compliance  with  USD(AT&L)  policy.  Obviously,  programs 
that  are  assessed  as  rating  on  the  low  end  of  the  OAAT  matrix  may  not  be  meeting 
the  intent  of  the  open  systems  approach  policy.  MDAs  should  be  asking  for  current 
OAAT  assessment  results  of  the  programs  during  the  major  milestone  program 
reviews. 

Finally,  as  more  programs  (weapon  systems  and  sub-systems)  throughout 
the  DoD  are  assessed  using  the  OAAT,  a  database  of  assessment  results  can  be 
established  as  a  repository  for  use  in  developing  lessons  learned  and  best  practices 
related  to  designing  systems  to  meet  a  higher  level  of  openness.  The  database  can 
be  organized  by  types  of  systems  (aircraft,  ship,  C4I,  space  ...)  to  be  used  in 
comparing  levels  of  openness  and  sharing  of  best  practices  and  lessons  learned  in 
achieving  those  levels  of  openness.  In  addition,  as  the  database  of  assessed 
programs  grows,  data  mining  techniques  can  be  used  to  compare  program 
acquisition  costs  and  their  relationship  to  higher  levels  of  openness.  This 
information  can  be  useful  in  making  determinations  of  desired  levels  of  openness, 
given  the  constraints  of  project  cost,  schedule,  and  performance. 
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VII 


Areas  for  Further  Analysis 


Although  this  research  effort  did  not  generate  sufficient  data  to  provide 
rigorous  analysis  of  the  assessment  results,  this  report  does  provide  a  preliminary 
conclusion  of  the  usefulness  of  the  OAAT  in  assessing  the  level  of  openness  of  DoD 
weapon  system  programs.  As  more  OAAT  assessment  results  are  generated  and 
added  to  this  research,  further  analysis  can  be  conducted  to  determine  any 
relationships  between  a  program’s  degree  of  openness  and  its  level  of  technology 
maturity,  phase  of  the  acquisition  lifecycle,  and  type  of  program  (non-developmental 
versus  developmental).  In  addition,  as  more  OAAT  assessment  results  are 
generated  and  added  to  this  research,  further  analysis  can  be  conducted  to 
determine  any  relationships  between  degrees  of  openness  for  weapon  system 
platforms  (such  as  aircraft,  ship,  and  ground  vehicle)  versus  sub-systems  (such  as 
navigational,  communication,  avionics,  and  propulsion).  With  this  additional  data, 
higher  granularity  and  more  rigorous  analyses  can  be  conducted. 

Finally,  it  would  be  beneficial  to  analyze  the  effectiveness  of  the 
implementation  of  MOSA  in  Navy  acquisition  programs  by  investigating  three 
different  facets  of  MOSA-based  programs:  1)  The  results  of  OAAT  assessments  of 
specific  weapon  system  acquisition  programs,  2)  The  status  of  USD(AT&L)  MOSA 
policy  compliance  as  presented  in  Milestone  Decision  Authority  (MDA)  review  briefs, 
and  3)  The  results  of  award-fee  determinations  on  OAAT -assessed  programs.  This 
further  analysis  should  specifically  compare  the  results  of  OAAT  assessment  results 
on  specific  DoD  acquisition  programs  with  the  status  of  MOSA  implementation 
reported  to  MDAs  (Reference  USD(AT&L)  memo,  5  April  2004,  requiring  “all 
programs  subject  to  milestone  review  shall  brief  their  program’s  MOSA 
implementation  status  to  the  Milestone  Decision  Authority  to  determine 
compliance.”)  and  with  the  results  of  contract  award-fee  determinations  on  MOSA- 
based  programs  that  have  award-fee  incentives  in  their  contracts.  (Award  fees  were 
identified  as  a  characteristic  of  successful  MOSA-based  contracts  in  the  initial 
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MOSA  research.  See  Rendon,  2005).  The  purpose  of  this  additional  analysis  would 
be  to  determine  if  the  OAAT  assessment  results  are  being  provided  to  the  MDA 
during  milestone  reviews,  and  also  to  determine  if  weapons  system  acquisition 
programs  assessed  at  higher  degrees  of  openness  are  paying  higher  award  fees  to 
contractors.  Thus,  this  further  analysis  would  provide  insight  into  the  implementation 
of  MOSA  in  acquisition  programs  by  comparing  the  status  of  MOSA  compliance  in 
MDA  review  briefs,  the  results  of  internal  MOSA  assessments  on  those  Navy 
acquisition  programs,  and  the  results  of  contract  award-fee  determinations  for  those 
acquisition  programs;  it  would  also  determine  if  any  conclusions  can  be  made  on  the 
effectiveness  of  MOSA  implementation  in  Navy  acquisition  programs.  The  results  of 
this  research  will  prove  beneficial  to  senior  Navy  officials  by  providing  data  points  on 
MOSA  implementation  by  analyzing  the  consistency  of  MOSA  compliance  status, 
assessments,  and  award-fee  results  for  specific  Navy  acquisition  programs. 
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VIII 


Conclusion 


The  purpose  of  this  research  was  to  analyze  the  results  of  past  OAAT 
assessments  of  selected  Navy  weapon  systems  and  sub-systems.  The  original 
vision  was  that  the  OAAT  assessment  results  would  be  widely  accessible  from  the 
domain  program  offices  to  support  this  research.  However,  since  only  two  Navy 
domains  had  responded  to  a  request  for  OAAT  assessment  results,  this  research 
focused  specifically  on  the  Subsurface  Domain  and  Air  Domain  programs,  and 
generally  on  how  the  OAAT  assessment  results  could  be  used  in  supporting  the 
DoD’s  policy  for  using  an  open  systems  approach. 

The  preliminary  research  on  the  assessment  results  for  the  Sub  and  Air 
Domain  programs  indicated  that  approximately  half  of  the  total  assessments 
conducted  for  both  the  Subsurface  and  Air  Domains  were  rated  at  the  High  level  of 
openness.  Fourteen  of  the  total  assessments  were  rated  at  the  Medium  level  of 
openness,  and  two  assessments  were  rated  at  the  Low  level  of  openness. 

Based  on  these  general  assessment  results,  it  can  be  seen  how  the  Open 
Architecture  Assessment  Tool  (OAAT)  can  be  used  to  effectively  assess  a  program’ 
(aircraft  or  system)  level  of  openness.  In  addition,  using  the  OAAT,  program 
managers  can  determine  which  areas,  Business/Programmatic  or  Technical,  need 
additional  emphasis  in  order  to  increase  the  program’s  level  of  openness.  The 
OAAT  can  also  be  effective  in  developing  a  road-map  for  a  program’s  strategy  for 
achieving  a  specific  level  of  openness  for  the  weapon  system  or  sub-system.  The 
OAAT  can  be,  and  should  be,  used  by  Milestone  Decision  Authorities  (MDAs)  in 
assessing  the  program’s  compliance  with  current  DoD  Open  Systems  policy. 

Finally,  it  is  proposed  that  a  database  of  assessment  results  be  established  as  a 
repository  for  use  in  developing  and  sharing  lessons  learned  and  best  practices 
related  to  designing  weapon  systems  and  sub-systems  to  meet  a  higher  level  of 
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■  USAF  IT  Commodity  Council 

■  USMC  Contingency  Contracting 
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Financial  Management 

■  Acquisitions  via  leasing:  MPS  case 

■  Budget  Scoring 

■  Budgeting  for  Capabilities  Based  Planning 

■  Capital  Budgeting  for  DoD 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Financing  DoD  Budget  via  PPPs 

■  Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition 
Budgeting  Reform 

■  PPPs  and  Government  Financing 

■  ROI  of  Information  Warfare  Systems 

■  Special  Termination  Liability  in  MDAPs 

■  Strategic  Sourcing 

■  Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 

Human  Resources 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 

■  Learning  Management  Systems 

■  Moral  Conduct  Waivers  and  First-tern  Attrition 

■  Retention 

■  The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 

■  Tuition  Assistance 

Logistics  Management 

Analysis  of  LAV  Depot  Maintenance 
Army  LOG  MOD 
ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
Diffusion/Variability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 
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■  Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

■  Optimizing  CIWS  Lifecycle  Support  (LCS) 

■  Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance 
Activity 

■  Pallet  Management  System 

■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

-  RFID  (6) 

■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  Aegis  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  Aegis  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Eared  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 


A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our 
website:  www.acquisitionresearch.org 
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